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Background / Context: 

Description of prior research and its intellectual context. 

Retell protoeols are eommonly used for assessing reading eomprehension (Cohen, Krustedt, & 
May, 2009) and, when ineluded with ORF measures, eontribute additional information about 
students’ reading eomprehension abilities (Mareotte & Hintze, 2009; Reed, Vaughn, & Petseher, 
in press). However, there is wide variability in the prompting eonditions. Prior researeh suggests 
alterations in the wording of a prompt require different eomprehension skills and result in 
different retell responses (Gagne, Bing, & Bing, 1977). In addition, there is disagreement as to 
whether oral or silent reading is more effieaeious for retell performanee (e.g.. Hale et ah, 2007, 
MeCallum, Sharp, Bell, & George, 2004; Miller & Smith, 1990; Prior & Welling, 2001). 

Purpose / Objective / Research Question / Focus of Study: 

Description of the focus of the research. 

The purpose of this study was to improve the utility of retell protoeols by more elearly 
identifying the testing eonditions under whieh students demonstrate better performanee. The 
researeh questions eoneern whether the wording of the initial prompt, the inelusion of a follow- 
up prompt, or the opportunity to silently re-read the passage is related to retell performanee. 
Beeause retells were seored quantitatively by the pereent of pre-determined idea units ineluded, 
it was hypothesized a prompt asking students to tell everything they remember would be 
assoeiated with better seores than a prompt asking students to tell what a passage is “mostly 
about.” Furthermore, it was hypothesized that students of various ability levels would have 
improved retells if provided an opportunity to re-read a passage silently prior to retelling it, 
rather than delivering their responses immediately following a 1 -minute timed ORF eomponent. 

Setting: 

Description of the research location. 

Students were from two middle sehools in Texas serving students in grades 6-8. Approximately 
94% were tested during their English language arts elass. 

Population / Participants / Subjects: 

Description of the participants in the study: who, how many, key features or characteristics. 

The original sample ineluded 589 sixth-, seventh-, and eighth-graders of all ability levels who 
returned parent permission forms to partieipate in the study. Seventeen of those students were 
absent or otherwise unavailable on the days of testing, and 45 students experieneed testing 
abnormalities as explained in the seetion on fidelity of test administration proeedures. This left a 
final sample of 527 students: 150 in grade 6, 168 in grade 7, and 209 in grade 8. Half of the 
students reeeived free or redueed-priee luneh, and 68% were Hispanie. 
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Intervention / Program / Practice: 

Description of the intervention, program or practice, including details of administration and duration. 



Students were randomly assigned to one of four retell testing conditions: 1) one-time oral reading 
and prompted to tell in their own words what the passage was mostly about; 2) one-time oral 
reading and prompted to tell everything they remembered; 3) one-time oral reading, prompted to 
tell everything they remembered, and provided follow-up prompting; 4) oral reading followed by 
silent rereading, prompted to tell everything they remembered, and provided follow-up 
prompting. The first condition was intended to capture an alternative way of asking for a 
retelling that would not specifically preference summarizing or giving a main idea (“what the 
passage was mostly abouf ’) and that would leave open the possibility of paraphrasing (“in your 
own words”). The latter was done to create more equivalency to the prompt used in the other 
three conditions, “tell me everything you remember reading in the passage.” Similarly, the “tell 
everything” prompt avoided using the word “retell,” so that the comparison between groups 
could more confidently be based on nuanced variations in prompt wording rather than on 
student’s level of familiarity with a named skilled (i.e., summarizing, identifying a main idea, 
retelling). The fourth condition introduced silent reading after the oral reading required for the 
ORF component of the assessment. This was done for two reasons: 1) having a silent reading 
only condition would require that three additional passages be administered, thus potentially 
confounding results with measurement artifacts; and 2) combining oral and silent reading would 
allow for a comparison between conditions 3 and 4 that was not based on an advantage of one 
method of reading over another, but on the efficacy of allowing more time to reread for meaning. 



Research Design: 

Description of research design (e.g., qualitative case study, quasi-experimental design, secondary analysis, analytic 
essay, randomized field trial). 

Randomized field trial 

Data Collection and Analysis: 

Description of the methods for collecting and analyzing data. 

The Passage Reading Fluency (ORF and retell components) and Word Reading Fluency sub tests 
of the Texas Middle School Fluency Assessment Assessment ([TMSFA], Texas Education 
Agency, University of Houston, & The University of Texas System, 2008) were administered in 
May at the End of Year testing point. The transcribed retell responses were scored by the 
principal investigator and two research assistants who had participated in the development of the 
scoring guides. Each response was compared to a set of pre-determined idea units for the 
passage. The scoring guides include allowable alternatives for wording the idea units, and scores 
are based on the percent of idea units accurately recalled out of the number possible for the word 
count attained. That is, students read to different places in the passage during the ORE portion of 
the assessment and, therefore, were eligible to recall a different number of idea units based on 
the total number of words read. The scoring guides provide the minimum word count needed to 
have read the information contained in each idea unit(see Appendix A for a sample scoring 
guide). 
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All data were manually entered into a database by the two research assistants involved in the 
development of the retell scoring guides. This provided another opportunity to check the 
accuracy of the scores recorded on the examiner documents. The research assistants also typed 
all student retells into the database before scoring them a second time to calculate inter-rater 
reliability. When entering the retell, assistants verified the number of words in the retell as 
recorded by the original examiner. Twenty-five percent of the retell responses were double 
scored for the purposes of determining inter-rater reliability among the three individuals applying 
the scoring guides. Using the database with the typed retells, an individual other than the original 
scorer applied the scoring guide to the student response a second time. Observed agreement on 
scores for individual passage retells ranged from a low of 82% (sixth-grade passage 1) to a high 
of 95% (seventh-grade passage 2). However, the average of the scores obtained on the three 
passages is the main unit of analysis for the TMSFA. Therefore, agreement was also calculated 
by determining the correlation of the average retell scores obtained by the two raters. The 
relationship, as measured by the intraclass correlation (i.e., .976), was strong and suggested 
agreement would not have occurred by chance. 

Means on each outcome measure were calculated for the full sample. In addition, means for the 
average retell score and the length of retell utterance were calculated for each testing condition. 
These were analyzed using one-way analysis of variance and Cohen’s d (Cohen, 1988). The 
Benjamini-Hochberg correction (Benjamini & Hochberg, 1995) was utilized to control for the 
false-discovery rate, rather than the family-wise error rate. 

Findings / Results: 

Description of the main findings with specific details. 

Because attrition in randomized experimental designs may create differences among the groups, 
it was important to assess the extent to which attrition contributed to a bias of estimated 
treatment effects. This was examined through a comparison of the differential proportion of 
missing data between the treatment and control groups with the overall proportion of missing 
data. Using guidelines outlined by What Works Clearinghouse (2008), the relationship between 
total missing and differential missing data for each outcome was determined to be at an 
acceptable level of bias under conservative assumptions. Students retold, on average, only about 
31% of the idea units they read in the passage. Their responses were about 33 to 38 words in 
length. The ORF portion of the Passage Reading Fluency subtest was strongly correlated (r = 
.861) to the Word Reading Fluency subtest, an ORF assessment of words in isolation. However, 
retell scores bore little relationship to ORF scores. With the exception of the weak correlation 
between retell scores on passage 2 with the Passage Reading Fluency equated score, only the 
lengths of utterances in retelling were significantly (r = .185 to .300) and mostly weakly related 
to the ORF components. This suggests retell is measuring different reading skills. 

Both the type of prompt and the use of follow-up prompting were significantly related to the 
percentage of pre-determined idea units retold. A change to either the wording or the addition of 
follow-up prompting produced approximately moderate effect sizes {d = .44 - .62). The 
combination of changes produced strong effect sizes {d = .96 - 1.05). However, the addition of 
silent reading did not significantly improve performance. Similar results were found when 
comparing the mean lengths of utterance. That is, students said significantly more words when 
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prompted to “tell everything” and when eneouraged to continue with follow-up prompting, but 
length of utterance was not significantly different when provided an opportunity to silently 
reread the passage before retelling on passages 2 and 3. On passage 1, the easiest of the three 
passages read, student retells were modestly though significantly longer when allowed to silently 
reread. 

Conclusions: 

Description of conclusions, recommendations, and limitations based on findings. 

When scored quantitatively, an initial prompt that asks students to tell everything they remember 
results in better responses than telling what the passage was “mostly about.” Moreover, 
encouraging students to continue retelling by asking if they remembered anything else, also 
significantly increases students’ scores. Offering addition time to reread a passage silently does 
not significantly improve retell as compared to retelling immediately after an ORF assessment. 

The results were similar when considering the number of words students included in their retells. 
The length of utterance was significantly longer when asked to tell everything and was longer 
still when encouraged to keep retelling. It seems students considered the initial prompt a literal 
indicator of what they were to do; either give a succinct “gist” of what the passage was “mostly 
about” or give as much information as possible. Yet even with the “tell everything” prompt, 
students did not provide all the information they actually recalled unless they were repeatedly 
prompted to continue retelling. 

These are important results given the great amount of variance in the prompts used across retell 
instruments (Reed, manuscript under review; Reed & Vaughn, manuscript under review). 
Attempts to synthesize or interpret retell scores from different studies will be problematic if there 
is not a movement toward greater consistency in the wording and delivery of prompts. An 
equally compelling finding was the lack of significant differences in the scores of students 
provided identically worded prompts and follow-up prompting, but a difference in the method of 
reading prior to retelling. Because retell is being included with ORF progress monitoring 
instruments (Good & Kaminski, 2010), adding just one minute onto the testing time for each 
passage can impact the feasibility of using frequently administered measures. The lack of strong 
correlations between retell and the ORF components of the TMSFA suggest retell is measuring 
some other skill than what is being captured by ORF (Marcotte & Hintze, 2009; Reed, Vaughn, 

& Petsher, in press). Therefore, retell has the potential to better inform reading instruction if 
administered under conditions that preserve instructional time and optimize student performance. 
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Appendices 

Not included in page count. 



Retell Scoring Guide 

Each row of the table represents an idea unit and is worth 1 point. 



Laura 

(page 1 of 3) 



f of 

Words 

Read 


O Recalled Idea Unit 


O 

Maximum 
Possible 
Idea Units 


7 


A girl 1 was a 1 "'If 
Aladv 1 ! 

Laura | y^fote children's books 
1 


1 


24 


She i . j 

, 1 was born 

Laura ; 1 


in a log house 
in Wisconsin 
in 1867 


2 


37 


I was close to her older sister 
5he i ^>3d an older sister named Mary 


3 


i i Charies 

1 was the second daughter of \ 


65 


They 

Her family 


had to rr 


1 sober j dad | j find 1 

lovp i wherever their | father j j get j 


4 


1 because her dad didn't have | 


72 


They 

Her family 


moved 


1 after Laura was born 
1 to Missouri 


5 


82 


They 

Her family 


moved 


1 to start a farm 
1 to Kansas 

1 where her sister Carrie was born 


6 


105 


They 

Her family 


moved 


1 back to Wisconsin 
; to be near family 


7 


122 


They 

Her family 

j 


moved 


1 in 1874 
1 to Walnut Grove 
1 to Minnesota 

J r\r 


8 


Her dad/father 1 wanted | ^ , 

, 1 , . to farm 

Charles longed 


151 


The girls | 

The kids j 

Laura and her sisters j 


had many adventures 


9 


176 


1 1 

1 1 

Grasshoppers | ate 1 their crops 

Bugs 1 destroyed | their wheat crop 

■ ■ 


10 
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Appendix B, Tables and Figures 

Not included in page count. 



Table 1 



Descriptive Statistics of Full Sample 





N 


Mean 


SD 


Retell 1 Score 


527 


30% 


20% 


Retell 2 Score 


527 


29% 


22% 


Retell 3 Score 


527 


35% 


26% 


Ave Retell 


526 


31% 


16% 


Retell 1 Word Count 


527 


38.17 


17.70 


Retell 2 Word Count 


527 


32.63 


16.23 


Retell 3 Word Count 


527 


37.33 


18.44 


PRF 


527 


160.66 


34.12 


WRF 


527 


76.82 


20.91 


TAKS 


509 


802.02 


97.55 


Benchmark 


497 


87.49 


11.50 


Valid N (listwise) 


488 







Note. PRF = Passage Reading Fluency subtest of the TMSFA; WRF = Word Reading Fluency 
subtest of the TMSFA 

Table 2 



Correlations Among Measures 





1 


2 


3 


4 


5 


6 


7 


8 


1. Retell 1 Score 


















2. Retell 2 Score 


.219** 
















3. Retell 3 Score 


.321** 


.213** 














4. Ave Retell 


.694** 


.659** 


.764** 












5. Retell 1 Word 


.653** 


.271** 


.289** 


.554** 










Count 


















6. Retell 2 Word 


.407** 


.518** 


.255** 


.542** 


.660** 








Count 


















7. Retell 3 Word 


.376** 


.296** 


.543** 


.582** 


.548** 


.554** 






Count 


















8. PRF 


-.066 


-.095* 


.008 


-.068 


.300** 


.250** 


.213** 




9. WRF 


-.074 


.002 


.000 


-.032 


.297** 


.276** 


.185** 


.861** 



**p< .01 

*p< .05 
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Table 3 



Descriptive Statistics by Condition 



isure 


N 


Condition 1 
Mean 


SD 


N 


Condition 2 
Mean 


SD 


N 


Condition 3 
Mean 


SD 


N 


Condition 

Mean 


ell 1 


129 


20% 


18% 


151 


29% 


18% 


134 


30% 


20% 


113 


37% 


ell 2 


129 


19% 


19% 


151 


28% 


20% 


134 


30% 


20% 


113 


35% 


ell 3 


129 


26% 


25% 


151 


35% 


24% 


134 


40% 


30% 


113 


39% 


1 Retell 


128 


22% 


14% 


151 


30% 


14% 


134 


40% 


20% 


113 


37% 


ell 1 WC 


129 


27.86 


13.35 


151 


36.11 


15.18 


134 


42.4 


15.8 


113 


47.71 


ell 2 WC 


129 


24.86 


14.21 


151 


30.39 


12.71 


134 


35.8 


16.1 


113 


40.78 


ell 3 WC 


129 


29.06 


15.5 


151 


36.47 


15.46 


134 


41.8 


20.7 


113 


42.62 




129 


159.82 


39.58 


151 


162.11 


34.34 


134 


158.6 


29.3 


113 


162.13 


P 


129 


75.46 


22.59 


151 


76.56 


20.93 


134 


76 


19.6 


113 


79.7 



Note. PRF = Passage Reading Fluency subtest of the TMSFA; WRF = Word Reading Fluency 
subtest of the TMSFA 



Table 4 

One-way Analysis of Variance: Average Retell 





Sum of 












Sauares 




Mean Sauare 


F 




Between Groups 


1.882 


3 


.627 


28.156 


.000 


Within Groups 


11.633 


522 


.022 






Total 


13.515 


525 









Table 5 



One-way Analysis of Variance: Length of Utterance 







Sum of 
Squares 


df 


Mean Square 


F 


Sig. 


Retell 1 Word Count 


Between Groups 


26992.037 


3 


8997.346 


34.141 


.000 




Within Groups 


137828.247 


523 


263.534 








Total 


164820.284 


526 








Retell 2 Word Count 


Between Groups 


17349.081 


3 


5783.027 


24.939 


.000 




Within Groups 


121278.278 


523 


231.890 








Total 


138627.359 


526 








Retell 3 Word Count 


Between Groups 


14758.303 


3 


4919.434 


15.678 


.000 




Within Groups 


164103.906 


523 


313.774 








Total 


178862.209 


526 
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